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ABSTRACT 



The University of Akron's Reading Assessment 



Component (RAC) research study assessed the effectiveness of a 
computer-based literacy program, Principles of the Alphabet Literacy 
System (PALS) , as an alternative to classroom instruction for 
functionally illiterate adults living in Summit County, Ohio. The RAC 
study assessed four objectives: (1) development of a pretest and 
posttest model to assess PALS; (2) measurement of quantitative and 
qualitative learning increases; (3) comparison of learning increases 
of PALS students to those of students in a selected adult basic 
education (ABE) program over the same time period; and (4) comparison 
of PALS program costs to the costs of a selected ABE program. 
Pretest/posttest scores were analyzed for a sample of 23 adults. Two 
client characteristics identified successful PALS participants: adult 
learners who scored at least a 75 on the Slosson Intelligence Test 
and those who completed at least 4 years of formal schooling. Six 
recommendations were made: extrinsic motivations should be used to 
improve attendance and lower dropout rates; a family literacy program 
should be established; quantitative and qualitative measures should 
be used to evaluate clients; follow-up telephone interviews should 
collect information from past clients; projected grade-level 
increases in reading achievement should be reevaluated? and greater 
care must be taken to enroll adults in appropriate literacy programs. 
(15 references) (NLA) 
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Aduit Literacy: An Evaluation Of A Successful Program 

American workers must be properly trained to read and write if they are going to 
keep pace with the demands of today's jobs. In the workplace, new technologies and 
management styles are increasing the level of basic skills required of workers. 
However, an estimated 27 million adults in the United States can be considered 
functionally illiterate - their basic skills are below the fifth grade level. Most are 
unemployed, or are employed in entry-level, unskilled positions. (Project Literacy 
U.S., 1990). 

In order to improve basic reading and writing skills, human service agencies 
are beginning to empower clients to overcome their communication difficulties by 
providing literacy instruction using the Laubach System and Principles of Alphabetic 
Literacy System (PALS). Laubach uses a progressive series of reading textbooks and the 
support of program volunteers to tutor adult learners in reading. PALS combines the 
power of a personal computer, infowindow, and videodisc to teach functionally illiterate 
adults to read and write during three phases of a 100 hour classroom program. 

Specifically, the University's Reading Assessment Component (RAC) team 
compared Test of Adult Basic Education (TABE) scores of clients in the PALS program 
with clients in the Laubach program in order to assess adult literacy increases. In 
addition, RAC evaluated the success of just the PALS program in the County of Summit's 
Department of Human Services using multiple testing instruments and the "portfolio 
approach". ("Portfolio approach" is defined as the client's file folder of chronologically 
collected writings and test results.) The results of the PALS evaluation and Laubach 
comparison document literacy improvement by PALS clients following participation in 
this 5 month computer-based program at least with respect to a specific client profile 
on a short term basis. 

Purposes of the RAC Evaluation . The purpose of the RAC research study was to assess the 
effectiveness of a computer-based literacy program titled PALS as an alternative to 
classroom instruction for functionally illiterate adults living in Summit County. The 
four specific RAC assessment objectives included: (1) Develop an appropriate pretest 
and posttest model to assess a computer-based adult literacy program; (2) Determine 
both quantitative and qualitative learning increases in an adult literacy population; (3) 
Compare learning increases of PALS students to a selected ABE program over the same 
time period; and (4) Compare PALS program costs to the costs of a selected ABE 
program. 

RAC Evaluation Design . The evaluation was designed to measure the literacy results of 
the PALS program as objectively and accurately as possible. To accomplish this goal, 
RAC decided that quantitative and qualitative data would be collected using a portfolio 
approach and a control group. Thereafter, RAC determined that assessment of increases 
in reading ability would be based on pretest and posttest (a) TABE scores from Akron 
and Canton; (b) Ekwall Informal Reading Inventory scores (starting points were 
determined by administering the San Diego Quick Assessment Test); (c) Wide Range 
Achievement Test results; (d) Dolch Word List scores; (e) the Roswell-Chall Blending 
Test, and (f ) the Wepman Auditory Discrimination Test Finally, RAC specified that the 
Slosson Intelligence Test would be administered and client writing samples would be 
collected. 

Assessment Difficu lties. An unavoidable problem that arose during assessment was 
client attrition and drop-out from participation in the PALS literacy program. During 
this thirteen month research study a total of 58 adults were referred to the PALS Lab in 
the Summit County Department of Human Services. However, 32 clients or 55% of the 
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program's total population stopped attending the computer-based instructional sessions 
for such reasons as SSI, incarceration, transfer into another literacy program, 
relocation, and employment. 43% of the PALS clients ceased participating in the 
program for unexcused reasons. Other clients were permitted to leave the program, 
despite having completed numerous assessment pretests. 

Results of the RAC Evaluation . The demographic profile of the sample of 23 adults in 
Akron who were referred for literacy services during the twelve month study is 
summarized as follows: (a) The mean age was 34.7 years; (b) The oldest participant 
was 59 years of age; (c) The youngest client was 23 years old; (d) The group consisted 
of 1 2 females and 1 1 males (52% females and 47% males) for a total sample population 
of 23 adults; (e) The highest grade achieved by adults in this study averaged 9.7; (f) 
65% of the group reported that they had been enrolled in special education programs; 
(g) Of the 23 adults in this study, only 22% stated that they were currently taking 
medication; and (h) All participants were unemployed during the 100 hour PALS 
program. T-tests comparing demographic data to changes in pretest and posttest scores 
indicated that the demographic characteristics of age, gender, special education 
placement, and medication were not good predictors of improved literacy scores. 
However, the variable of highest grade completed does correlate with client discrepancy 
scores. 

Results of Assessment Objective 1: If there were no constraints operating with 
respect to test selection an ideal pretest and posttest model to assess a computer-based 
adult literacy program would likely include some of the following multiple testing 
instruments: (1 ) Achievement Tests - Adult Basic Learning Examination, California 
Achievement Test, Metropolitan Achievement Test, Nelson Denny Reading Test, Test of 
Adult Basic Education, Wide Range Achievement Test; (2) Criterion-Refer mce Tests - 
Stanford Diagnostic Reading Test, Woodcock Reading Mastery Test; (3) Intelligence Test 
- Individual - Slosson Intelligence Test, Stanford-Binet Intelligence Scales, Wechsler 
Adult Intelligence Scale; (4) Intelligence Test - Group - California Test of Mental 
Maturity, Culture Fair Intelligence Test; (5) Reading Survey and Diagnostic Tests - 
Bader Visual Discrimination Test, McCarthy Individualized Diagnostic Reading 
Inventory, Reading Miscue Inventory; (6) Screening Tests - Audiometer, Harris Test 
of Lateral Dominance, Keystone Visual Screening Tests. 

Results of Assessme nt Objective 2: The quantitative learning increases in the 
adult population tested in this study indicated that: (1 ) Only 1 3% of the PALS clients 
were reading at or above the 5th grade level before entering the program; (2) 87% of 
the participants in the study were functionally illiterate before PALS; (3) PALS clients 
were not performing to their capacity as reflected in reading expectancy scores; (4) 
41% of the PALS clients scored "adequate" on the Wepman Auditory Discrimination Test 
and 47% scored "adequate" on the Roswell-ChaH Blending Test Adequate scores on both 
tests reflect the strength of the learners phonics skills - the foundation of the PALS 
program; (5) Clients scored higher on all parts of the Ekwall (Silent) following their 
participation in the PALS program; (6) The higher the client's performance on the SIT 
the higher the individual's performance on the TABE and Ekwali posttests; (7) Small 
gains were made by PALS participants on the Dolch Test; (8) WRAT reading scores 
improved between pretesting and posttesting, however, spelling scores actually 
decreased slightly following enrollment in the PALS program; and (9) PALS clients 
experienced positive and significant improvements in TABE Vocabulary, Comprehension, 
and Total test scores. 

The qualitative teaming increases in the adult population surveyed in this study 
are reflected in client portfolios. The writing samples chronologically arranged in client 
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file folders and examined by RAC faculty revealed that during the course of the PALS 
program adult learners: (1) began to exhibit greater complexity in ideation; (2) gained 
greater control over mechanical processes such as punctuation and spelling; (3) 
expanded their writing repertoire from strictly personal narratives to include fetters 
and creative stories. 

Results of Assessment Objective 3: At the outset it should be noted that Dr. John 
Henry Martin and IBM recommend that adults should not be enrolled in a PALS program 
until they are able to satisfy specific entry requirements, including (Martin, 1 986): a 
standardized reading score at or below a fifth grade level, an IQ rating of 75 or above, an 
inability to write a simple sentence, and a lack of any physical handicaps that interfere 
with learning such as hearing loss, vision impairments, or brain damage. 

The "successful" PALS client in this study did experience approximately an 8 
month gain in reading ability during the 1 00 hour program, as reflected in post TABE 
Vocabulary, Comprehension, and Total scores. Vocabulary skills improved by 
approximately 8 months, comprehension by 1 0 months, and total skills increased 9 
months. While Akron PALS clients improved 9 months in total reading skills the Canton 
Laubach group increased their reading abilities by 1 year and 9 months. Thus, Laubach 
clients seem to have experienced a higher gain in reading achievement. T-tests revealed 
that the changes in TABE Total test scores for PALS and Laubach program participants 
were both positive and significant at p - .002. 

Results of Assessment Objective 4: The PALS program appears to have an 
initially beneficial effect. However, it is a costly program. There are two main 
expenses involved: personnel costs and equipment costs. Costs per client were derived 
for the PALS program from the following ingredients based on current market values and 
replacement costs for a twelve month period: (1) Salary plus fringe benefits; (2) 
Facilities based on the replacement cost of a classroom with a life of twenty years, an 
interest rate of 7.5%, and an annualization factor of .0989; (3) Equipment costs 
amortized over 60 months; (4) Average monthly materials and supplies costs; and (5) 
Miscellaneous average monthly costs for energy, maintenance, administrative overhead, 
insurance, and training. 

The effectiveness of the PALS instructional method was determined from the 
change in pretest and posttest TABE scores of PALS clients in comparison with those of 
the control group. When these results are combined with those for costs, a cost- 
effectiveness ratio was derived that showed a cost of $1 ,605.00 per PALS student for a 
one-month improvement in test scores (Levin, 1 983). 

Discussion of Results : As the pretest and posttest scores of the PALS and Laubach groups 
are studied, attention needs to focus on the fact that the Canton participants possessed a 
higher skill level in vocabulary and comprehension upon entrance into the program, as 
demonstrated by TABE Total scores. However, it was beyond the scope of this study to 
establish a causal relationship between higher prerequisite skills and two year literacy 
increases. Thus, inferences based only on TABE Total scores from the Canton control 
group without additional diagnostic testing, intelligence evaluation, and demographic 
information should be made with caution. 

The results of the study also need to be examined in light of the dearth of proven 
effective teaching and testing materials for adults. According to Metz in the Journal of 
Reading , "it appears that both standardized tests and competency-based assessment 
poorly serve the adult learner. Standardized tests are related to former failure in 
school, are intimidating, and give a one-sided view of a multiskted problem. Competency 
programs assess competencies imposed on a learner rather than the learner choosing 
competencies that match his/her goals" (p. 469). 
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In addition, this discussion must recognize the technology integrated in the PALS 
program. According to Soifer (1990), to be literate in today's world, people must feel 
comfortable using computer technology. The PALS program seeks to accomplish this goal 
by integrating technology and learning. Adults participating in the program learn 
valuable reading and writing skills during the 100 hour program while acquiring 
technological training using the word processor that may one day make them self- 
supporting. 

Conclusions 

Profile : This research study, assessing adult literacy gains following participation in 
the PALS program, appears to identify specific client characteristics that contributed to 
an individual's overall success in the program. The client characteristics limited to the 
sample population in this study and determined by regression analysis are profiled in 
the following conclusions: Conclusion 1. The "successful" PALS participant was an adult 
learner who scored at least a 75 on tfie SIT; Conclusion 2. The "successful" PALS 
participant was an adult learner who completed at least 4 years of formal schooling. 

Although the profile of "successful" PALS client is not a stable model because of 
the small sample size, the profile does serve to demonstrate that a learning community 
is being built by the Summit County Department of Human Services that is empowering 
clients to take an active role in determining their own future. In this effort, people are 
beginning to examine the assumptions that underlie the way in which "service" is defined 
by and for them. If learning increases and adult literacy improvements are going to 
continue through organized classes by human service agencies then it is imperative that 
clients referred to the PALS program come equipped with the prerequisite skills 
identified by Dr. John Henry Martin and IBM in order to experience lifelong learning and 
literacy success. 

Collaboration : Collaboration was a motivating force behind this research project. 
Specifically, The University of Akron's College of Education and the Summit County 
Department of Human Services entered into their first contractual arrangement for 
professional services. In addition, the conceptual framework for the project 
represented a collaborative research effort by representatives form the College's 
departments of Elementary Education, Counseling and Psychology, and Special Education. 
The project was organized and executed in conjunction with numerous planning 
meetings, telephone conversations, and site visits. 

Throughout the study, administrators, faculty, staff, and clients endeavored to 
cooperate fully with respect to the time and effort demands of the research design. 
Friendships were fostered, experiences were shared, and a level of frustration was 
experienced as a result of the testing, reporting, and interviewing schedules required by 
the study. Moreover, participants began to realize the political needs and overtones of 
the project results. 

Based on the experiences of the participants, the following guidelines are 
proposed to help ensure productive collaborative projects for future teams: (1 ) 
Collaboration teams should be composed of professionals who are interested in group, not 
individual, research; (2) Team members should share a common philosophy with 
respect to assessment methods and evaluation techniques; (3) Project coordinators 
should identify clear, concise, feasible goals for the project that accurately reflect the 
available resources; (4) All collaboration participants should commit themselves to 
working within a specific timelir*; (5) Team members should be required to attend 
regular meetings in order to enhance communication throughout the entire project; (6) 
Monthly budget reports should be distributed to all team members so that financial 
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expenditures can be monitored; and (7) Politicalization should be minimized by 
insulating team members from the influences of persuasive forces. 

Recommendations 

(1) As Balmuth (1989) recognizes, high rates of absenteeism and drop-out 
plague ABE programs. However, research indicates that effective recruiters including 
former PALS students, reading teachers, and reading tutors can be trained to attract 
program participants and support their regular attendance. Extrinsic motivation, such 
as incentive pay and food coupons, may be increased on the basis of attendance and 
achievement 

(2) Nickse, Speicher, and Buchek (1988) report in the Journal of Reading that 
literacy programs aimed at parents and their children have proven to be successful for 
both groups. The Akron Departments Human Services may, therefore, consider 
sponsoring a literacy program that seeks to break the illiteracy cycle of individuals and 
families by encouraging entire families to attend PALS classes. 

(3) In addition, evaluating PALS participants with both qualitative and 
quantitative measures by implementation of the portfolio approach proved to be a helpful 
strategy. Folders were kept on each client so that diagnosticians and Lab staff were able 
to study the data from more than just one standardized test and examine a progressive, 
chronological series of writing samples. 

(4) The Summit County Department of Human Services may continue to evaluate 
the effectiveness of the PALS program by collecting valuable follow-up information 
from past clients by telephone. Calls should be initiated several weeks after the 
beginning of the literacy instruction and could be used to determine whether the 
participant was satisfied with the initial instructional experience, and reasons, if any, 
for non-completion or dissatisfaction. Program administrators may then be able to 
determine weaknesses and remedy problems in the program. 

(5) The RAC team also recommends that the testing division of IBM associated 
with the PALS program re-evaluate the projected grade level increases in reading 
achievement expected for adult learners in a 100 hour literacy program. As stated 
earlier in this report, this period is equivalent to just 3 weeks of daily instruction in a 
high school. This 1 00 hour program extended over as much as a nine month period for 
many of the parti:ipants studied in this project. Rate of retention needs to be considered 
when learning is spread out over such an extended period of time. 

(6) Finally, it is important to recognize that greater care must be taken to 
enroll adults in the appropriate literacy program. Many of the individuals receiving the 
services of the Summit County Department of Human Services would likely benefit from 
enrollment in a ff pre-PALS M and/or a "post-PALS" program. Clients who do not satisfy 
the PALS entry requirements identified by Dr. John Henry Martin and IBM and 
supported by this research study may appreciate the opportunity to develop or refresh 
their communication skills by participating in a slower paced, tutorial program with 
computer drill and practice exercises before entering PALS. 
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